Real-World Material Recognition for Scene Understanding

In this paper we address the problem of recognizing materials in consumer photographs. While material recognition isn’t a new problem, the introduction of the OpenSurfaces dataset [1], allows it to be studied at a new scale. In particular, the dataset provides materials in a huge variety of real-world environments, with dramatic appearance and shading differences within each a material class. We propose a discriminative learning framework for the per-pixel classification of materials in an image. Huge appearance variation makes classifying some material classes extremely challenging our method achieves only 34.5% classification accuracy. However, we show that even this weak material signal can be valuable for scene understanding. We use the output of our classifier as a new feature in a recent RGB-D scene understanding algorithm. We improve this state-ofthe-art scene understanding method by 0.7%.